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Introduction 

Almost  everyone  has  heard  the  term  simulation ,  but  who 
knows  what  it  really  means?  Is  simulation  represented  by  a 
wargaming  effort,  such  as  Pacific  Air  Forces  (PACAF)  units 
participating  in  a  TEAM  SPIRIT  exercise?  Or,  is  a  better 
example  provided  by  a  C-5  aircrew  practicing  emergency 
procedures  in  a  cockpit  mockup?  Perhaps  simulation  is  best 
demonstrated  by  somebody  analyzing  a  computer-based 
model  to  determine  the  expected  number  of  backorders  of 
some  reparable  item  for  the  Joint  Strike  Fighter? 

Whatever  simulation  means,  the  military  seems  to  think  it 
is  worthwhile.  A  few  examples:  Joint  Vision  2010 
specifically  cites  simulation  as  a  method  of  improving 
training  realism,  promoting  readiness  and  assessing 
operations  concepts.1  The  DoD  Directive  for  defense 
acquisition  requires  that  “modeling  and  simulations  shall  be 
used  to  reduce  the  time,  resources,  and  risks  of  the  acquisition 
resources.”2  Finally,  the  DoD  has  established  a  modeling  and 
simulation  master  plan3  and  an  entire  organization  to  address 
simulation  issues.4 

In  fact,  simulation  includes  wargaming,  training  and 
analysis.  It  is  generally  defined  as  a  modeling  process 
whereby  entities  (that  is,  objects  of  interest — which  can 
include  real  people,  machines  or  even  failure  or  repair 
actions)  interact  in  a  defined  way,  over  a  period  of  time.  The 
terms  modeling  and  simulation  are  often  used  interchangeably; 
however,  this  is  not  really  correct.  A  model  is  simply  an 
approximate  representation  of  some  piece  of  our  world.  A 
model  can  be  either  physical  (as  in  a  miniature  wooden  replica 
of  an  aircraft)  or  symbolic  (as  in  the  mathematical  equation 
of  distance  as  a  product  of  velocity  and  time).  Simulation  is 
merely  one  method  of  building  and  using  a  model.  For 
example,  other  ways  of  building  and  analyzing  symbolic 
models  include  the  operations  research  optimization 
techniques  of  linear  and  nonlinear  programming.  Baker  and 
Grabau  discuss  the  modeling-simulation  distinction  in  more 
detail  in  a  recent  issue  of  Program  Manager.5 

Simulation  is  used  when  other  methods  are  too  dangerous, 
too  expensive  or  impractical.  In  a  wargaming  exercise,  it  is 
safer  to  pretend  that  someone  is  the  enemy  instead  of 
engaging  a  real  one  and  to  use  laser  gear  or  paintballs  instead 
of  real  munitions.  Since  we  don’t  yet  have  operational 
experience  with  the  Joint  Strike  Fighter  (JSF),  symbolic 
model  simulations  are  very  useful  for  estimating  JSF  support 
requirements.  Incidentally,  we  generally  use  simulation  for 
symbolic  models  only  when  these  models  cannot  be  solved 
by  analytic  means.  This  is  because  simulations  typically  give 
us  only  approximate  solutions  instead  of  exact  values. 
Furthermore,  it  is  difficult  to  use  simulation  to  optimize  a 
model’s  input  values. 

The  remainder  of  this  article  provides  an  overview  of 
simulation,  with  emphasis  on  logistics  modeling.  Key  DoD 


simulation  agencies  are  introduced.  The  critical  area  of 
verification  and  validation  is  discussed,  and  the  article 
concludes  with  recommendations  for  further  reading. 

A  Simulation  Taxonomy 

Neyland  identifies  three  commonly  used  simulation 
categories:  live ,  virtual  and  constructive .6  Live  simulation 
is  the  process  of  real  people  using  real  machines  while 
pretending  to  perform  some  activity,  instead  of  actually  doing 
it.  The  TEAM  SPIRIT  example  falls  into  this  category.  Virtual 
simulations  still  involve  real  people,  but  now  they  are  using 
mockups  instead  of  real  equipment.  A  classic  example  is  an 
aircrew  using  a  cockpit  mockup,  as  in  the  C-5  example. 
Finally,  constructive  simulations  consist  of  models  of  people 
and  machines.  Constructive  simulations  are  typically 
accomplished  by  running  a  symbolic  model  on  a  computer. 
An  example  would  be  to  run  the  Logistics  Composite  Model 
(LCOM) — a  powerful  tool  that  is  generally  used  to  identify 
the  best  mix  of  logistical  resources  to  support  a  given  weapon 
system  under  operational  constraints.7 

Real  Time 

A  key  distinguishing  characteristic  between  live,  virtual 
and  constructive  simulations  is  the  passage  of  time.  Live  and 
virtual  simulations  both  use  real  time — one  second  on  a  wall 
clock  is  equivalent  to  one  second  of  simulation  time.  In  a  live 
simulation  such  as  a  RED  FLAG  exercise,  commanders  and 
operators  are  able  to  affect  the  course  of  the  simulation  by 
periodically  making  decisions  or  taking  action  and  then 
observing  the  effect  of  those  actions.  In  contrast,  constructive 
simulations  typically  use  either  expanded  or  compressed 
time — one  second  on  a  wall  clock  could  be  the  same  as  either 
a  nanosecond  or  a  year  of  simulation  time.  For  example,  in 
just  a  few  wall  clock  minutes,  an  analyst  can  use  LCOM  to 
simulate  five  years  of  base-level  aircraft  support  activity.  The 
problem  is  that  after  a  constructive  simulation’s  initial 
conditions  and  runtime  constraints  are  specified,  little  or  no 
human-model  interaction  is  possible  until  the  simulation  run 
is  complete.  In  our  LCOM  model,  for  example,  we  cannot 
arbitrarily  hit  the  computer  pause  key  sometime  during  a 
simulation,  pretend  an  enemy  just  blew  up  a  back-shop  and 
then  resume  the  simulation  to  see  what  happens.  Instead,  both 
the  enemy  attack  and  some  feasible  workaround  strategies  for 
the  missing  back  shop  would  need  to  be  scripted  in,  before 
the  simulation  begins. 

A  Common  Technical  Framework 

Imagine  that  we  have  access  to  two  virtual  simulators — a 
desktop  computer-based  MiG  23  program  and  a  $10M  F-15C 
motion-base  dome  simulator.  Could  we  connect  the  two 
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systems  and  let  ‘em  battle  it  out?  Several  issues  arise:  do  the 
two  simulations  use  the  same  standards  for  describing  and 
sharing  data?  Can  we  synchronize  the  clock  timing  of  the  two 
simulations?  Do  the  two  simulations  share  a  common 
perception  of  the  battlespace  and  of  each  other’s  respective 
weapon  system  capabilities?  The  idea  of  linking  a  desktop 
computer  simulator  to  a  motion-base  dome  system  may  seem 
extreme,  but  in  reality  wargamers  and  others  are  increasingly 
interested  in  the  ability  to  network  live,  virtual  and  even 
constructive  simulations  into  a  single  effort  (that  is,  into  a 
system  of  systems).  If  the  individual  simulations  could  be 
truly  interoperable,  then  the  limitations  of  any  single 
component  simulation  should  be  transparent  to  the  others. 
For  example,  neither  our  MiG  23  pilot  nor  our  F-15C  pilot 
should  be  able  to  tell  whether  the  other  is  flying  a  dome  or  a 
desktop  simulator.  The  main  difference  for  the  two  pilots 
should  only  be  in  the  amount  of  realism  each  experiences  in 
the  simulated  battle.8  The  need  to  resolve  interoperability 
issues  and  promote  the  reusability  of  simulations  led  the  DoD 
to  establish  a  common  technical  framework  (CTF),  to  which 
individual  simulation  efforts  must  conform.  The  CTF  is  a 
product  of  the  Defense  Modeling  and  Simulation  Office 
(DMSO),  located  in  Alexandria,  Virginia.9  The  CTF  consists 
of  three  parts: 

•  A  high  level  architecture  (HLA),  which  is  a  set  of 
conceptual  rules  and  specifications  that  prescribe  how  the 
different  simulations  will  work  together. 

•  A  conceptual  model  of  the  mission  space  (CMMS) 
which  is  essentially  a  common  understanding  of  what  the  real 
world  looks  like. 

•  A  set  of  data  standards,  which  includes  things  like 
physical  data  representation,  data  quality  and  data  security.10 

Hollenbach  and  Alexander  use  the  analogy  of  city  planning 
to  illustrate  the  CTF  concept,  noting  that 

...  to  build  and  operate  an  efficient  city,  a  governing  framework 
(for  example,  street  plans,  building  codes,  ordinances)  is  laid 
out  and  certain  basic  services  (for  example,  utilities,  schools, 
fire  protection)  are  provided.  Beyond  that  the  residents  are 
generally  left  to  their  own  discretion  as  to  what  type  of  home 
or  business  they  build,  who  they  interact  with  .  .  .  .n 

Systems  of  Systems 

Military  analysis  can  benefit  from  considering  virtual 
systems  of  systems.  In  November  1997,  the  Army  investigated 
the  denial  of  global  positioning  system  data  on  battallion¬ 
sized  operations  by  linking  four  M1A1  Abrams  tank 
simulators  at  the  Simulation  Network  (SIMNET)  facility  at 
Fort  Hood,  Texas,  with  two  helicopter  simulators  at  Fort 
Rucker,  Alabama,  and  a  fuel  truck  simulator  at  Fort  Knox, 
Kentucky.  This  virtual  simulation  enabled  the  Army  to 
predict  how  soldiers  could  use  new  technologies  under  a 
variety  of  conditions.12 

Wargaming  activities  frequently  use  a  suite  of  virtual 
simulation  models.  A  commonly  used  simulation  is  a  two- 
sided  theater  air  campaign  model  called  Air  Warfare 
Simulation  Mode  (AWSIM).  It  is  typically  used  to  train  battle 
staffs  and  also  acts  as  a  nonscripted  command  post  exercise 
driver.  Another  example  is  the  corps  battle  simulation  (CBS). 
CBS  simulates  both  air  and  ground  forces  and  is  used  for  battle 


staff  training.  Neither  AWSIM  nor  CBS  can  model  all  aspects  of 
a  campaign  and  so  both  are  frequently  used  in  conjunction  with 
other  simulations  during  an  exercise  (for  example,  AWSIM  does 
not  model  space-based  systems  or  information  warfare).13 


A  principal  shortcoming  of 
wargaming  simulations  is  that 
they  do  a  poor  job  of  modeling 
logistics. 


Logistics  Realism 

A  principal  shortcoming  of  wargaming  simulations  is  that 
they  do  a  poor  job  of  modeling  logistics.  For  example,  a 
documented  problem  with  AWSIM  is  its  inadequate 
representation  of  air/ground  mobility  and  resupply, 
maintenance,  personnel  and  non-weapon  consumption 
rates.14  In  recognition  of  this  shortcoming  and  to  help  train 
staff  logisticians,  the  Headquarters  United  States  Air  Forces 
in  Europe  Deputy  Chief  of  Staff  for  Logistics  (HQ  USAFE/ 
LG)  established  the  Logistics  Exercise  Enhancement  Program 
(LEEP).  LEEP  consists  of  a  Logistics  Simulation  (LOGSIM) 
model  for  replicating  base-level  logistics  and  a  program  of 
documentation  and  training  support  for  injecting  logistics 
realism  into  wargaming  exercises.  LOGSIM  has  supported 
the  last  three  USAFE  UNION  FLASH  exercises  and  HQ 
PACAF’s  ULCHI  FOCUS  LENS  98.15  Another  wargaming 
simulation  initiative — the  Joint  Simulation  System  (JSIMS) — 
hopefully  will  increase  logistics  emphasis.  The  JSIMS  goal 
is  to  be  “the  primary  modeling  and  simulation  tool  to  support 
future  joint  and  Service  training,  education,  and  mission 
rehearsal.”16- 17 

Constructive  Logistics  Models 

Over  the  years,  logistics  has  probably  benefited  more  from 
constructive  simulation  modeling  (such  as  LCOM)  than  from 
virtual  simulation  efforts.  One  reason  is  because  constructive 
simulations  are  particularly  useful  for  analysis.  Logistics 
problems  are  frequently  too  difficult  to  solve  by  analytic 
methods.  For  example,  monthly  demand  for  spare  parts  is 
typically  random.  This  demand  uncertainty  makes  an 
inventory  problem  much  harder  to  solve  than  an  inventory 
problem  with  constant  demand  and  when  an  analyst  must 
determine  optimal  stocking  levels  and  reorder  points  for 
many  parts  at  once,  the  problem  becomes  enormous.  RAND’s 
Dynamic  Multi-Echelon  Technique  for  Recoverable  Item 
Control  (Dyna-METRIC,  version  6)  was  one  of  the  early 
popular  simulation  models  developed  for  Air  Force  problems 
in  reparable  inventory  theory.18  Previous,  analytic  versions 
of  METRIC  were  based  on  steady-state  conditions  that 
precluded  modeling  dynamic  factors  like  wartime  surges  in 
aircraft  usage  rates  and  uncertain  support  capabilities.  Dyna- 
METRIC  can  handle  these  dynamic  factors  while  also 
accommodating  lateral  resupply — which  is  very  difficult  to 
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capture  in  an  analytic  inventory  model.  The  gains  in  modeling 
flexibility  from  simulation  do  not  come  for  free,  however. 
Because  Dyna-METRIC  is  a  simulation,  it  cannot  optimize  spares 
requirements  to  achieve  specific  goals. 19  A  more  recent  inventory 
analysis  simulation  example  is  provided  by  the  Defense  Logistics 
Agency’s  (DLA)  Performance  and  Requirements  Impact 
Simulation  (PARIS)  model.  PARIS  is  used  to  examine 
investment,  inventory  and  supply  chain  policies  for  the  more  than 
1 .9  million  spare  parts  that  DLA  manages.  It  can  simulate  two 
years  of  demand  on  190,000  items  in  under  two  hours.20 

Inventory  problems  are  certainly  not  the  only  logistics 
problems  that  benefit  from  simulation.  LCOM  is  widely  used 
to  address  a  variety  of  base-level  logistics  issues  including 
sortie  generation  rates,  personnel  requirements  and  aircraft 
availability.21  For  example,  the  F-22  System  Program  Office 
is  now  using  LCOM  to  estimate  sortie  generation  rates  and 
maintenance  personnel  requirements.22 

Transportation  theory  can  also  benefit  from  simulation 
modeling.  Air  Mobility  Command  (AMC)  uses  the  Airlift 
Flow  Model  (AFM) — a  simulation  model  embedded  within 
the  Mobility  Analysis  Support  System — to  assess  policies  for 
airlift  control,  mission  planning  and  mission  execution.  The 
AFM  provides  a  global  airlift  simulation  of  AMC  and 
commercial  airlift  assets  in  strategic  and  theater  operations. 
AFM  can  simulate  airborne  refueling,  aircrews  and  their 
flying  hour  limits  and  all  phases  of  aircraft  ground  handling. 
Since  its  development,  no  serious  airlift  analysis  has  been 
performed  without  at  least  comparing  the  results  with  output 
from  AFM.23 

Constructive  Simulation  Tools 

Throughout  the  1980s,  the  suite  of  available  simulation 
tools  was  pretty  limited.  Personal  computers  were  not  very 
powerful  and  few  commercial  simulation  software  packages 
existed.  The  analyst  was  mostly  limited  to  writing  simulations 
in  a  general  purpose  language  (such  as  PASCAL  or 
FORTRAN  [Formula  and  Translation]),  and  running  the 
models  on  a  mainframe  computer.  The  result  was  that 
simulations  tended  to  be  difficult  to  build  and  maintain  and 
were  seldom  interoperable.  Things  have  changed 
dramatically  in  the  last  ten  years.  Today  over  40  different 
commercial  simulation  software  packages  are  available.24  The 
tremendous  improvements  in  personal  computer  hardware 
help  analysis  as  well.  For  example,  DLA’s  PARIS  simulation 
was  built  using  AweSim™  (a  commercially  available 
simulation  program)  and  runs  on  an  NT®  workstation.  PARIS 
replaces  a  mainframe  computer-based  FORTRAN  model  that 
was  difficult  to  maintain  and  change.25 

Some  constructive  simulations  can  even  be  run  on  a 
standard  personal  computer  spreadsheet!  Monte  Carlo 
simulations  are  those  in  which  time  has  no  real  relevance  to 
the  problem.  Instead,  the  goal  is  to  determine  the  outcome 
of  a  series  of  random  experiments.  For  example,  a  gambler 
does  not  really  care  how  long  it  takes  to  play  a  series  of  poker 
hands.  The  important  aspect  is  whether  the  gambler  wins  or 
loses  each  game.  If  we  were  to  simulate  the  reliability  of  an 
aircraft  landing  gear  over  a  series  of  landings,  we  would  be 
more  interested  in  the  landing  gear’s  failure  history  than  on 


how  long  each  landing  takes.  Spreadsheets  can  readily 
accommodate  Monte  Carlo  simulations,  especially  when  a 
spreadsheet  add-in  such  as  Decisioneering’s  Crystal  Ball  or 
Palisade’s  @Risk  is  used.  Dyna-METRIC  is  a  (non¬ 
spreadsheet  based)  Monte  Carlo  simulation. 

Simulation  Model  Credibility 

No  discussion  of  simulation  is  complete  without 
addressing  model  verification  and  validation  (V&V).  In  fact, 
this  topic  is  so  important  that  the  DoD  issued  Defense 
Instruction  5000.61,  DoD  Modeling  and  Simulation 
Verification,  Validation,  and  Accreditation ,  in  April  1996. 
The  Air  Force  also  has  guidance,  found  in  AFI 16-1001, 
Verification,  Validation,  and  Accreditation,  dated  June  1996. 
Verification  seeks  to  address  whether  we  have  built  our  model 
right  (does  our  model  satisfy  our  design  requirements?),  while 
validation  focuses  on  whether  we  have  built  the  right  model 
(are  our  design  requirements  themselves  correct?).26 

The  real  goal  of  V&V  is  to  get  the  principal  users  of  a 
simulation  model  to  feel  confident  about  it.  The  outcome  of 
a  V&V  effort  is  not  a  yes/no  answer — there  is  no  such  thing 
as  absolute  validity.  We  usually  do  not  have  enough  time  or 
money  to  check  every  aspect  of  a  model.  Finally,  there  is  no 
such  thing  as  general  validity — a  model  that  is  valid  for  one 
purpose  may  not  be  valid  for  another. 

Symbolic  models  are  typically  verified  by  using  standard 
computer  programming  debugging  techniques.  Example 
methods  include  building  and  checking  a  model  in  logical 
chunks,  starting  with  a  simple  model  and  adding  complexity 
only  as  needed  and  ensuring  that  units  of  measurement  are 
consistent.  Sometimes  a  simulation  model  can  be  simplified, 
and  its  output  compared  with  an  analytical  result. 

Validation  is  generally  harder  to  perform  than  verification. 
Validation  asks:  how  does  the  simulation  model  compare  to 
reality?  We  want  the  model  to  be  good  enough  to  use  in  the 
same  way  we  would  use  the  real  system.  This  implies  that  a 
model’s  assumptions  and  limits  must  be  clearly  defined  and 
documented,  else  we  risk  using  the  model  under  conditions 
that  render  it  invalid. 

A  key  V&V  goal  is  to  develop  a  simulation  model  with  high 
face  validity.  This  means  that  the  model  and  its  output  seem 
reasonable  to  experts  in  the  field.  A  typical  validation 
method  is  to  compare  a  model’s  output  to  historical  data  (if 
available)  or  to  the  output  from  a  similar  simulation  model. 
For  example,  DLA  compared  their  PARIS  model  results  to 
output  from  the  model  it  replaced,  because  changes  in  policy 
and  demand  patterns  made  it  impractical  to  compare  PARIS 
output  to  historical  data.27  Another  way  to  boost  face  validity 
is  to  involve  the  model’s  eventual  users  continuously 
throughout  the  model’s  development.  Regular  involvement 
is  a  great  way  to  promote  user  buy-in  to  the  overall  effort  and 
ensures  that  nobody  is  surprised  over  what  the  final 
simulation  model  can  or  cannot  do. 

Finally,  note  that  ease  of  validation  depends  on  the  actual 
system  of  interest.  It  would  be  straightforward  to  validate  a 
simulation  model  of  some  aspect  of  a  current  depot’s 
operation,  because  the  real  operation  is  an  existing, 
observable  process.  Now  imagine  how  we  would  validate  a 
model  of  on-equipment  maintenance  in  the  proposed  Space 
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Station.  The  best  we  could  probably  do  would  be  to  compare 
our  model’s  output  to  data  from  Skylab  or  the  Russian  MIR 
program. 

Simulation  Education 

At  least  one  course  on  simulation  is  offered  in  many 
graduate  schools,  including  most  civilian  universities,  the  Air 
Force  Institute  of  Technology  (AFIT)  and  the  Naval 
Postgraduate  School.  AFIT  requires  that  a  simulation  course 
be  taken  by  students  in  logistics  masters  degree  programs.  A 
typical  simulation  course  teaches  the  constructive  modeling 
aspect  of  simulation — for  some  reason,  little  emphasis  is 
placed  on  live  or  virtual  simulation.  Law  &  Kelton28  and 
Banks,  Carson  and  Nelson29  represent  the  two  most  popular 
textbooks  on  constructive  simulation  modeling.30 


Virtually  every  military  member  is 
likely  to  participate  in  or  be 
affected  by  a  simulation  effort 
during  the  course  of  a  career. 


Conclusion 

Virtually  every  military  member  is  likely  to  participate  in 
or  be  affected  by  a  simulation  effort  during  the  course  of  a 
career.  Why?  Simulation  helps  us  get  the  most  benefit  from 
our  defense  dollars.  We  can  use  simulation  to  help  warfighters 
train,  to  develop  doctrine  and  to  perform  analysis  on  almost 
any  military  topic.  An  increasing  need  exists  to  be  able  to 
integrate  live ,  virtual  and  constructive  simulations  into 
systems  of  systems.  The  DMSO’s  common  technical 
framework  is  the  key  initiative  that  will  make  these  systems 
of  systems  feasible.  Finally,  we  must  never  depend  on 
simulation  modeling  as  a  complete  replacement  for  working 
with  reality.  Real  systems  contain  subtleties  and  uncertainties 
that  our  models  will  probably  never  capture  completely. 
However,  simulation  can  give  us  useful  insights  at  a  fraction 
of  the  cost  and  risk. 
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